Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create env files from templates in a change-preserving way #5099

Merged
merged 2 commits into from
Oct 31, 2024
Merged

Conversation

dhruvkb
Copy link
Member

@dhruvkb dhruvkb commented Oct 28, 2024

Fixes

Fixes #2938 by @stacimc

Description

This PR updates the env recipe to make it "smarter". Instead of copying over the file and overwriting changes, the Python script now keeps changed env values and even extra env key-value pairs in the file.

Assume the following env.template file:

key_one=value_one
key_two=value_two

Assume the following .env file:

key_one=new_value_one
key_three=value_three

After just env the .env file would be as follows.

key_one=new_value_one
key_two=value_two

# Preserved variables
key_three=value_three

Testing Instructions

  1. Make some changes to the catalog/.env file. You could change some values, delete some keys or add some new keys.
  2. Run just env.
  3. See that the changes are preserved, deleted keys are restored and custom extra keys are kept at the bottom of the file.
  4. Confirm this against the printed logs.

Shortcomings

This approach does not preserve any extra comments in the .env file. Those will be deleted.

Checklist

  • My pull request has a descriptive title (not a vague title likeUpdate index.md).
  • My pull request targets the default branch of the repository (main) or a parent feature branch.
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no visible errors.
  • I ran the DAG documentation generator (ov just catalog/generate-docs for catalog
    PRs) or the media properties generator (ov just catalog/generate-docs media-props
    for the catalog or ov just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@dhruvkb dhruvkb requested a review from a team as a code owner October 28, 2024 14:39
@dhruvkb dhruvkb requested review from krysal and obulat and removed request for a team October 28, 2024 14:39
@openverse-bot openverse-bot added 🟩 priority: low Low priority and doesn't need to be rushed 🧰 goal: internal improvement Improvement that benefits maintainers, not users 🤖 aspect: dx Concerns developers' experience with the codebase 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Oct 28, 2024
Copy link
Member

@krysal krysal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works great, cool!

@AetherUnbound
Copy link
Collaborator

This approach does not preserve any extra comments in the .env file. Those will be deleted.

Is there any way to preserve those comments without ballooning this code? 😅 I've often left stuff commented out (like variables that would reference production resources) for testing and it would be ideal to have those retained. Perhaps the script could make a .env.backup-YYYY-MM-DD file as well which is added to the .gitignore, but will preserve the old file in its entirety?

@dhruvkb
Copy link
Member Author

dhruvkb commented Oct 29, 2024

@AetherUnbound that's interesting! I feel like a backup would be simpler but it does need one to compare the old and new file and manually move any keys that were changed or removed. At that point I'm not sure there is that much difference between diffing the new .env with its backup vs. not using this script and diffing the current .env with the template.

Are you suggesting the backup as an addition/improvement over this PR or does the use-case for preserving comments occur frequent enough to invalidate this PR?

@dhruvkb dhruvkb merged commit c41e110 into main Oct 31, 2024
43 checks passed
@dhruvkb dhruvkb deleted the smart_env branch October 31, 2024 05:11
@AetherUnbound
Copy link
Collaborator

Thanks for adding that backup creation step! That's exactly what I was thinking 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🤖 aspect: dx Concerns developers' experience with the codebase 🧰 goal: internal improvement Improvement that benefits maintainers, not users 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Add just recipe to update catalog .env
4 participants